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ABSTRACT 

Higher education administrators often wonder if the 
information they receive from the low response rates to surveys of 
non^matriculat ing applicants accurately reflects the entire 
population of these non-matriculants. These questions are exacerbated 
by evidence that response bias does exist. This paper describes how 
St. Mary's College in Notre Dar^;, Indiana, studied its response bias 
through mail and phone follow-up surveys. Results are reported from: 
(1) two mailed surveys, one sent to all accepted applicants who 
notified the institution of their intent to attend elsewhere and 
another shorter version (postcard) mailed to those vho had not 
responded to the first survey; and (2) a final telephone follow-up to 
all remaining non-respondents. Findings from these efforts showed few 
differences between those who responded to the initial survey and 
those contacted in the subsequent surveys. It was concluded that 
there is no need to adjust for non-respondent bias in future surveys. 
The cost effectiveness of telephone follow-up, particularly when 
coupled with its high response rate, suggests that future surveys of 
non-respondents should' util ize this method exclusively, rather than 
the postcard survey. (GLR) 
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WHO DOESN'T RESPOND TO APPLICANT SURVEYS? 
AN ANALYSIS OF RESPONDENT BIAS 

Introduction 

In an era of ever-increasing competition for students in a declining pool of traditional- 
age prospects, colleges and universities have stepped up their meurketing efforts to seek ways 
to attract a larger segment of this population. Hossler (1986, p. 43) suggests that admissions 
personnel need to thoroughly understand what institutions are their chief competitors in order 
to devise strategies to lure applicants into their institution. Davis- Van Atta and Carrier 
(1986, p. 82) note that the most important step in the enrollment decision for a student is 
choosing one school. Market research can provide the admissions personnel with a list of the 
institution's primary competitors and the applicants' reasons for selecting these schools. One 
common tool for determining market information such as the institutional image and major 
competitors is the mail survey of applicants. 

Because response rates are usvaily quite low on these surveys, particularly for the 
non-matriculants, whose responses are especially desired, the question of whether or not there 
is a response bias is often raised. When bias is detected using information already available 
(such as test scores and application information), it suggests that other biases may exist 
between the respondent group and the non-respondents. Light, Singer, and Willett (1990, p. 
69) advise that if nonresponse exists, the best techniques for measuring nonresponse bias are 
callbacks, sampling nonrespondents, and the use of nonrespondents from earlier surveys as 
replacements for people who are nonrespondents in a current survey. 
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This paper presents the results of a study which sought to identify the important biases 
between the respondents and the non-respondents on a survey of non-matriculating accepted 
applicants. A second, but different, follow-up mail survey and a telephone follow-up 
gathered information from those who had not responded to previous surveys. The 
methodology of these surveys is presented, along with recommendations based on this 
particular study that shoulu interest others contemplating similar research. 

Saint Mary's College has been surveying its non-matriculating applicants for a number 
of years in an effort to discover as much as possible about the factors involved in the decision 
to choose another college or university, with the purpose of strengthening its own marketing 
strategies. Saint Mary's, a Catholic college for women, is a competitive, highly ranked 
comprehensive college in the liberal arts tradition. The 1,500-1,600 women enr ^Ued are 
primarily residential students. While many of these women come from the Midwest region, 
the College attracts applicants from every state and many foreign countnes. 

In previous years, it had been noticed (see Table 1) that respondents to the non- 
matriculating survey tended to have higher scores on their SATs or ACTs than did the entire 
non-matriculating group as a whole. These respondents also tended to have better high 
school academic records (CPAs and class rank) than did the entire group'. There also 
appeared to be a higher response rate from applicants whose home states were closer to the 
College. 

As administrators reviewed the results of these surveys and noted the generally low 
response rates (shown also on Table 1), they wondered about the validity of the findings. 
Was it possible that this bias was systematic, and there were important non-matriculant 
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Previous evidence of response bias 
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High School 







SAT-Verbal 


SAT-Math 


GPA 




Resp 










Resp All 


Year 


Rate 


Resp 


All 


Resp 


All 


1987 


39.6% 


565 


545 


521 


505 




1988 


40.1% 


566 


547 


531 


508 




1989 


39.8% 


561 


549 


518 


507 




1990 


37.7% 


564 


548 


528 


501 




1991 


34.S% 


579 


547 


539 


511 


3.48 3.40 


1992 


40.1% 


571 


541 


513 


496 


3.34 3.36 



subgroups who were not responding proportionately. Were the non-respondents more likely 
to be attending a particular type of college or university? Were these non-matriculants from 
diffe ethnic and religious backgrounds? Was the College losing potential students for ^ 
whom appropriate modifications in the current marketing strategy might make a difference in 
their final decision as to which college to attend? Because there was an apparent response 
bias, it was decided to do a follow-up of tlie survey respondents in order to find out more 
about this group and to determine the degree of bias. 

Methodology 

An initial one page (front and back) survey was mailed by the Admission Office to all 
accepted applicants who notified the institution of their intent to attend elsewhere. A month 
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later, a second and shorter survey was mailed to all nonrespondents from the Office of 
Institutional Research. This second survey, consisting of a few items printed on a prepaid 
postcard, was purposely kept brief to encourage response. The cover letter for each survey 
suggested different reasons for requesting information. At the end of the summer, phone 
calls were placed to all remaining nonrespondents, requesting a few items of information. 
For logistic reasons, the few Internationa' subjects were excluded from this study. 

Comparisons were made between various respondent groups on variables such as 
academic abilities (ACT/SAT scores, high school rank and CPAs), personal characteristics 
(ethnicity and religious affiliation— the student body at Saint Mary's is predominantly Roman 
Catholic), choices of post-secondary institutions (categorized as prime competitor. Catholic 
competitors, private competitors, and public competitors), home and college locations (by 
geographic region), time of application, and time of cancellations. The criterion variables for 
these analyses were the various response categories (appropriate combinations of non- 
respondent and survey, card, and/or phone respondents). 

Results 

The response rate for the initial two-page survey was 23.7% (actually, there was a 
40.1% response, but 16.4% were anonymous and were not analyzed to avoid duplication of 
information) and the response rate for the card follow-up was 25. 1%. An additional 40.4% 
were contacted through the telephone survey. Combining all three forms, information was 
obtained from nearly 90 percent of the population. 

Goodness of fit tests were made using appropriate statistical techniques. Where the 
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variable being tested was categorical, chi-square analysis was used. Where the variable type 
was interval, t-tests or ANOVAs were utilized. Table 2 shows the summary of the results of 
these bivariate comparisons. Despite the preliminary evidence that there appeared to be 
response bias in previous years, the present data shows very little evidence of systematic bias. 

Significant differences (p < .05) were found only for the choice of college, the 
ranking of Saint Mary's among competitor colleges and universities, ethnicity, region of 
home, the time of cancellation, and high school rank. No significant differences (p < .05) 
were found on the basis of the type of college selected, the region of the selected college, 
applicant's religious affiliation, the time of application, ACT/SAT scores, and high school 
GPA. 

Looking at the areas in which response bias did occur, it was found (see Table 3) that 
non-matriculated applicants who considered Saint Mary's to be their first-choice college were 
more likely to respond to the initial survey than were those that ranked Saint Mary's lower on 
their list of preferred institutions. 

The card and telephone survey follow-up was more successful in the states closest to 
the College and less successful in the states further away (see Table 4). 

Further, the non-matriculated applicants who are m( .e likely to respond to the initial 
survey are not from ethnic minority groups (see Table 5). The minority applicants were also 
more difficult to contact by telephone as well. 

The time of notification of intended cancellation (see Table 6) was found to differ 
among the various respondent groupings, with those that cancelled earlier being more likely 
to respond to the initial survey. 
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Table 2 

Summary of bivariate comparisons 







Survey 














vs. 


Survey 










Survey 


Card 


vs. 










vs. 


vs. 


Card/ 


Survey 


Card/ 






Card 


Phone 


Phone 


vs. 


Phone 


Survey 




vs. 


vs. 


\/c 
Va. 


Corf]/ 


VQ 

TO* 






Phone 


Non-Resp 


Non-Resp 


Phone 


Non-Resp 


Non-Resp 


College type 


NS' 


NA' 


NA 


NS 


NA 


NA 


SMC 1st choice 


.008 


NA 


NA 


.013 


NA 


NA 


SMC rank 


.031 


NA 


NA 


.018 


NA 


NA 


Home region 


NS 


NS 


NS 


NS 


.017 


NS 


College region 


NS 


NA 


NA 


NS 


NA 


KA 


Ethnicity 


NS 


.002 


.001 


NS 


.001 


NS 


Religion 


NS 


NS 


NS 


NS 


NS 


NS 


Apply date 


NS 


NS 


NS 


NS 


NS 


NS 


Cancel date 


NS 


.031 


NA 


NS 


NS 


NS 


SAT-Verbal 


NS 


NS 


NS 


NS 


NS 


NS 


SAT-Math 


NS 


NS 


NS 


NS 


NS 


NS 


Enhcd ACT Comp 


NS 


NS 


NS 


NS 


NS 


NS 


HS RaJik 


NS 


NS 


NS 


NS 


.034 


NS 


HS GPA 


NS 


NS 


NS 


NS 


NS 


NS 



< .05; chi-square used for categorical variables; t-tests or ANOVA used with variables having 
ratio/interval scores 

^A indicates that test not possible due to unavailability of data from non-respondents 
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Table 3 

Choice and ranking of Saint Mary's College by respondent categories 





Survey 


Card 


Phone 


Survey 


Card/ 
Phone 


SMC 1st choice 


38.0% 


15.5% 


46.5% 


38.0% 


62.0% 


Not 1st choice 


23.3% 


31.7% 


45.0% 


23.3% 


76.7% 


Total 


26.6% 


28.1% 


44.3% 


26.6% 


73.4% 


Chi-square = 9.72006 
df = 2 
p = .0078 
N = 320 






Chi-square = 5.41705 
df = 1 
p = .0199 
N = 320 




Survey 


Card 


Phone 


Survey 


Card/ 
Phone 


SMC 1st choice 


38.0% 


15.5% 


46.5% 


38.0% 


62.0% 


SMC 2nd choice 


25.4% 


29.5% 


45.1% 


25.4% 


74.6% 


SMC 3rd or higher 


18.8% 


30.7% 


50.5% 


18.8% 


81.2% 


Total 


26.2% 


26.6% 


47.3% 


26.2% 


73.8% 


Chi-square = 10.60952 
df = 4 
p = .0313 
N = 294 






Chi- 
df = 

P = 
N = 


square = 8.02980 
= 2 

.0180 
= 294 



The final item shown on Table 2 showing a significant difference (p < .05), was 
based on the differences in high school rank between the non-respondents and those who 
responded to either the card or pho..e follow-up. This result was inconsistent with the other 
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Table 4 

Response rates for non-matriculating applicants based on geographical region of home 

Non- Card/ 
Resp Phone 

More distant' 19.4% 80.6% 

Closed 3.9% 96.1% 

Prime Midwest' 9.0% 91.0% 

Total 10.7% 89.3% 

Chi-square = 8.19353 
df = 2 
p = .0166 
N - 262 

'New England, South, Southwest, and West 
^Middle states. Midwest 
'Illinois, Indiana, Michigan, Ohio 



analyses based on academic ability and background, and is likely an artifact due to large 
differences in group size (39 vs. 235) and unequal variances. 

Because of the ease in which the telephone survey was conducted and the greater 
success in obtaining useful information regarding the applicant using this method, a cost 
analysis of the two different follow-up methods was done in order to ascertain whether this 
particular method was most cost-effective. The results of this analysis are shown in Table 7. 
In addition to these direct costs, there was the indirect expense of labor involved in preparing 
the mailings, making the telephone calls (about 20 hours were spent in this endeavor), and 
processing the results. 
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Table 5 

Response rates for non-matriculating applicants based on ethnicity 



Non- 







Resp 


Survey 


Card 


Phone 


Non-White 




24.2% 


14.5% 


24.2% 


37.1% 


White 




8.1% 


25.6% 


25.3% 


41.1% 


Total 




10.9% 


23.7% 


25.1% 


40.4% 


Chi-square 
df = 3 
p = .0017 
N = 359 


= 15.13849 


Non- 
Resp 


Survey 


Card/ 
Phone 




Non-White 




24.2% 


14.5% 


61.3% 




White 




8.1% 


25.6% 


66.3% 




Total 




10.9% 


23.7% 


65.5% 




Chi-square 
df = 2 
p = .0005 
N = 359 


=/ 15,11333 


Non- 
Resp 


Card/ 
Phone 






Non-White 




28.3% 


71.7% 






White 




10.9% 


89.1% 






Total 




14.2% 


85.8% 







Chi-square = 9.27248 
df = 1 
p = .0023 
N = 274 
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Table 6 

Response rates based on date of admission cancellation 





Non- 
Resp 


Survey 


Card 


Phone 


Before April i 


15.0% 


30.0% 


25.0% 


30.0% 


Betw Apr 1 & May 1 


5.3% 


25.2% 


29.8% 


39.7% 


Betw May 1 & June 30 


12.8% 


21.2% 


25.6% 


40.4% 


After July 1 


18.8% 


21.9% 


3.1% 


56.3% 


Total 


10.9% 


23.7% 


25.1% 


40.4% 



Chi-square = 18.35196 
df = 9 
p = .0313 
N = 359 

Discussion 

The findings that there were so few biases evident between the various respondent 
groups was surprising, particularly in light of previous evidence that some bias was probably 
present. While these results are valid only for this particular survey group, it provides 
evidence that it is possible that response bias is limited in such survey applications. 

The bias found related to choice and ranking of Saint Mary's College was not felt to 
be particularly damaging to the analysis of the results obtained by the surveys. The 
applicants who consider Saint Mary's College to be their first choice are those whom the 
College would prefer to have as students; hence their decision not to matriculate and their 
reasons for this decision are more important than had they not been as interested in attending 
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Table 7 



Cost analysis of follow-up methods 



Attempts 



Contacts 



Cost' 



Cost/ 
Contact 



Post Card Survey 



275 



90 



$162 



$1.80 



Telephone Survey 



165 



145 



$ 70 



$0.48 



*Costs include printing, postage, and telephone charges; labor not included 

the College and had applied to Saint Mary's as their second (or lower) choice. 

The results related to the home region of the applicants are probably due to the greater 
difficulty in contacting these persons at times when they were home and that were convenient 
to the telephone survey, factors arising from the different time zones involved. 

The findings related to the ethnicity of the applicants concern the College, particularly 
since it is making a strong effort to improve the diversity of the student body. Why these 
applicants decide to attend another college or university is crucially important to the analysis 
of the marketing strategy used particularly for this group of applicants. That so many of 
these fail to respond provides only minute evidence for the review of present strategies. 
Further study may need to be given to this group in subsequent years. 

The higher response rate to the initial survey for those who had cancelled earlier is 
probably simply the result of having a longer time to respond to the survey. Also, these 
applicants received their survey earlier in the process, and had perhaps not yet been inundated 
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by the follow-up surveys of other institutions; hence, tliey were more likely to cooperate with 
the request from Saint Mary's College. 

The cost analysis showed that the telephone survey is nearly three times as efficient in 
terms of direct cost per contact. Not only is there little expense for the non-respondents, but 
it is possible to obtain valuable information from parents and/or siblings even if the subject is 
not available. 

Conclusions 

This particular study found very little bias between those who responded to the initial 
survey and the subsequent surveys; hence there is little need to adjust for this bias in future 
surveys. The cost-effectiveness of telephone follow-up, particularly when coupled with its 
high response rate suggests that future surveys of non-respondents utilize this method 
exclusively, rather than the postcard survey. Finally, whether or not any bias is detected, it 
is recommended that follow-up studies of survey non-respondents be conducted periodically in 
order to validate (or adjust) the results obtained from a low-response group. 
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Endnotes 

1 . It should be noted that the SAT scores are self-reported for the respondent group. A 
cross-check of the validity of these self-reported scores in 1990 showed that they were quite 
accurate. In 1992, 95% of those providing scores on their survey were within 20 points of the 
SAT score submitted to the College. Where there was error, it usually was in reporting a higher 
score (though this may also be a result of retesting which had not been reported to the College); 
it was also noted that the students with lower scores tended to not report these scores. These 
trends probably account for much of the apparent bias in the SAT scores. The 1992 averages 
for these groups based on scores officially submitted to the College was 554 for the SAT-V and 
498 for the SAT-M; these figures are based on 85% of those who reported scores on the survey, 
as a number of these had not submitted SAT scores to the College. The high school grade-point 
average was not self-reported, but determined from information provided by high school 
transcripts and high school counselors. 
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WHO DOESN'T RESPOND TO APPLICANT SURVEYS? 
AN ANALYSIS OF RESPONDENT BIAS 

ABSTRACT 

Higher fjducation administrators often wonder if the information they receive from the 
low response rates to surveys of non-matriculating applicants accurately reflects the entire 
population of their non-matriculants. These questions are exacerbated by evidence that 
response bias indeed does exist. This paper describes how one institution studied its response 
bias through mail and phone follo^^^p surveys. It should be of interest to institutional 
research professionals whose institutions are interested in the characteristics of their non- 
matriculants. 
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